Comparison of Botometer vs. BotometerLite Bot Detection Scores

Abstract

BotometerLite is advertised as a lightweight bot detector that improves scalability by focusing on only user profile information; furthermore, BotometerLite claims using fewer features only entails a small compromise in individual accuracy. We test the validity of this claim by comparing Botometer with BotometerLite bot likelihood scores for 10,000 randomly sampled users. BotometerLite scores varied drastically from Botometer scores.

Introduction

• Training and performance evaluation described in “Scalable and Generalizable Social Bot Detection through Data Selection” Yang et al. (2020).
• BotometerLite released in September 2020.

Bot Type Scores

Bot scores describe how much an account acts like a specific kind of bot. https://botometer.osome.iu.edu/faq

• Astroturf: manually labeled political bots and accounts involved in follow trains that systematically delete content
• Fake follower: bots purchased to increase follower counts
• Financial: bots that post using cashtags
• Self declared: bots from botwiki.org
• Spammer: accounts labeled as spambots from several datasets
• Other: miscellaneous other bots obtained from manual annotation, user feedback, etc.
• Complete Automation Probability: the probability, according to the Botometer model, that an account with this score or greater is controlled by software.

Methodology

1. Randomly sample n accounts from Twitter API.
2. Collect Botometer and BotometerLite bot likelihood scores.
3. Calculate the correlation between scores.

Results

1. BotometerLite is most similar to the Botometer fake follower and spammer scores with \(R^2\) values of 0.394 and 0.334 respectively. Hence, if Botometer scores are accurate, BotometerLite may be somewhat effective at identifying potential fake followers and spammers.

The pearson correlation matrix (\(R^2\) values are the square of the values of this matrix) also shows the scores are weakly correlated.

#

Conclusion

References

Rauchfleisch, Adrian, and Jonas Kaiser. 2020. “The False Positive Problem of Automatic Bot Detection in Social Science Research.” Berkman Klein Center Research Publication, nos. 2020-3.

Yang, Kai-Cheng, Onur Varol, Pik-Mai Hui, and Filippo Menczer. 2020. “Scalable and Generalizable Social Bot Detection Through Data Selection.” In Proceedings of the Aaai Conference on Artificial Intelligence, 34:1096–1103. 01.